HLD Blueprint
The HLD Blueprint: 15 Components of a System Design
How to use this page: it's the checklist behind every design on this site. In a real 45–60 minute round you won't cover all 15 in depth. You'll cover 1–5 for every problem, then pick the 2–4 deep-dive components the problem is actually about (a chat app is about async processing and distributed ordering, a URL shortener about caching and ID generation, a payment system about distributed transactions and idempotency). Knowing all 15 is what lets you choose well and answer any follow-up.
Fundamentals this builds on: DNS · HTTP · Latency Numbers · Back-of-the-Envelope Estimation · SQL vs NoSQL · Indexes · ACID vs BASE · Caching Strategies · Cache Eviction · CDN
What interviewers are grading
Most system design interview rubrics come down to four signals:
| Signal | What it looks like |
|---|---|
| Problem navigation | You clarify scope, find what's actually hard, and prioritize instead of boiling the ocean |
| Solution design | A working end-to-end design first, then sensible components that fit the requirements |
| Technical depth | You can go deep where it matters: the failure modes, data structures and numbers |
| Trade-offs and communication | You state the alternatives and why you chose one, drive the conversation, and adapt to hints |
Seniority changes the bar. Mid-level candidates need a sound, working design. Senior and staff candidates are expected to drive the interview, spot bottlenecks unprompted, and reason about operations, evolution and cost.
Time budget (45-minute round)
| Phase | Minutes | Components |
|---|---|---|
| Requirements and scope | 4–6 | 1 |
| Estimation (if wanted) | 3–5 | 2 |
| API + data model | 5–8 | 3, 4 |
| High-level architecture (working end to end) | 8–10 | 5, 6 |
| Deep dives (interviewer-led or your pick) | 12–18 | 2–4 of 7–13 |
| Operations, trade-offs, evolution | 3–5 | 14, 15 |
Golden rule: get a simple design working end to end before optimizing anything. An elegant cache in front of a design with no write path is a fail.
1. Problem definition and requirements
Goal: turn a vague prompt ("design Instagram") into a bounded problem you can solve in 40 minutes.
Functional requirements: what the system does. List them, then prioritize the top 3–4 and say which ones are out of scope.
"Users can upload photos, follow users, and view a home feed. Out of scope: DMs, stories, ads, search."
Non-functional requirements: how well it does it. Put a number on each one:
| NFR | Questions to ask | Example |
|---|---|---|
| Scale | DAU? Read:write ratio? Data volume? Global? | 500M DAU, 100:1 reads |
| Latency | Which operations are latency-critical? What p99? | Feed p99 under 200 ms. Upload can be async. |
| Availability | What's the cost of downtime? | 99.99% for reads |
| Consistency | Can users see stale data? For how long? | Feed: eventual (seconds). Payments: strong. |
| Durability | Can we ever lose data? | Never lose an uploaded photo |
| Other | Privacy, compliance (GDPR, data residency), cost, mobile/offline |
Availability, translated:
| Availability | Downtime per year | Per month |
|---|---|---|
| 99% | 3.65 days | ~7.3 hours |
| 99.9% | ~8.8 hours | ~43.8 min |
| 99.99% | ~52.6 min | ~4.4 min |
| 99.999% | ~5.3 min | ~26 s |
Serial dependencies multiply: three 99.9% services in series give about 99.7%. Redundant parallel copies add nines: two independent 99% replicas give 1 − 0.01² = 99.99%.
Assumptions: say them out loud. "Assume an average photo is 200 KB after compression." "Assume authentication already exists."
Red flags: starting to draw boxes in minute 1. Asking 15 questions without converging. Never stating the NFRs, which are what decide the architecture.
2. Capacity estimation
Goal: turn the NFRs into numbers that justify decisions. The full method is in Back-of-the-Envelope Estimation.
Cover: DAU/MAU → QPS (average and peak) → storage (with retention and replication) → bandwidth (ingress and egress) → cache size → server count, and say what growth over 1–5 years does to each.
Always end with a design implication: "Peak is 70k reads/s, which is 100x the writes, so we optimize the read path with precomputed feeds and caching."
What bandwidth cost actually is: cloud providers almost universally charge for data egress — bytes leaving their network to the internet, or to another region — and not for ingress, which is usually free. The bill is GB transferred out × the provider's egress rate, and that rate is tiered: a free allowance, then a flat per-GB price for the next several terabytes, cheaper again at larger committed volume. Traffic to a CDN on the same provider, or between services in the same region, is typically cheaper or free, which is the other reason (besides latency) that co-locating chatty services in one AZ matters, and why CDN offload is usually the single biggest lever on this bill for a media-heavy system: a cache hit at the edge never touches your origin's egress meter at all.
3. API design
Goal: define the contract between clients and the system. It reveals the core entities and operations.
POST /v1/photos → 201 { photoId, uploadUrl } (Idempotency-Key header)
GET /v1/users/{userId}/feed?cursor=abc&limit=20
→ 200 { items: [...], nextCursor }
POST /v1/users/{userId}/follow → 204
GET /v1/photos/{photoId} → 200 { ... } (ETag for caching)
| Topic | What to say |
|---|---|
| Style | REST for public and mobile clients. gRPC between services. WebSocket/SSE for push. GraphQL for many heterogeneous clients. See HTTP §8. |
| Pagination | Cursor-based (an opaque token encoding the last seen key) over offset. Offset gets slower with depth (OFFSET 100000 scans and discards rows) and skips or duplicates items when data shifts. |
| Idempotency | Idempotency-Key on every non-idempotent write that clients might retry (payments, orders, posts). The server stores key → response with a TTL. |
| Errors | Consistent error bodies (e.g. RFC 9457 problem details), correct status codes, Retry-After on 429/503 |
| Large uploads | The client gets a pre-signed URL and uploads directly to object storage, so the bytes never pass through your app servers. Use multipart/resumable uploads for big files. |
| Long-running work | 202 Accepted plus a status resource or a webhook |
| Versioning | /v1/ in the path, or a header. Additive changes only within a version. |
| Auth and limits | Bearer tokens. Rate limits per user, API key and IP. |
| Concurrency | ETag + If-Match for optimistic concurrency on updates (412 on conflict) |
Red flags: verbs in URLs (/getUserFeed). No pagination on list endpoints. Exposing internal IDs or DB structure. Ignoring retries.
Going deeper on three rows above:
Large uploads, step by step: the client asks your app server for an upload slot; the app server generates a presigned URL (a short-lived, scoped link that authorizes one specific PUT to one specific object key) and returns it; the client uploads the bytes straight to object storage, never through your app servers; the client (or a storage event) then tells the app server the upload finished, so it can record the metadata. For anything too big for one PUT, a multipart upload splits the file into chunks (S3's default is up to 10,000 parts), uploads them — in parallel, if the network allows — and finishes with a single call listing every part's ETag. Resumable uploads build on the same idea: if a part fails, retry only that part instead of restarting the whole file.
Long-running work: the server validates the request, enqueues a job, and replies 202 Accepted with a Location header pointing at a status resource (/jobs/{jobId}) immediately — it never makes the client wait on the actual work. A worker pulls the job off the queue, processes it, and writes the result somewhere durable (a DB row or an object). The client either polls that status resource (pending → running → completed/failed) or gets pushed a webhook callback once it's done; a live-updating UI uses a WebSocket or SSE connection instead of polling. Give the job submission itself an idempotency key, the same as any other retried write, so a client that resubmits after a timeout doesn't start the work twice.
API versioning: URL path versioning (/v1/users, /v2/users) is simple, visible in every log line, and trivial to route at the load balancer or gateway — its cost is that you're running two full API surfaces side by side. Header versioning (Accept: application/vnd.myapi.v2+json) keeps one stable URL per resource, which REST purists prefer, at the cost of being invisible to anyone just reading request logs or testing in a browser. Either way, "additive-only within a version" means exactly this: a client built against v1 must keep working forever on v1 — you add new optional fields, you never remove or repurpose an existing one. Retiring an old version is its own process: announce a sunset date, add a Deprecation/Sunset response header, watch per-client usage to know who's still on it, and give a real grace period before the hard cutoff.
4. Data modeling
Goal: the entities, their relationships, and, most importantly, the access patterns that decide the keys and the storage engine.
Process:
- List the entities (User, Photo, Follow, FeedItem) and their key attributes.
- List the access patterns with their frequency: "get the feed for a user, newest first" (very hot), "get the followers of a user" (fan-out), "get a photo by ID".
- Choose the primary keys and partition keys to serve the hot patterns without scatter-gather.
- Decide normalize vs denormalize: normalize for write integrity, denormalize (or precompute) for hot reads.
- Normalization stores each fact exactly once. A user's display name lives only in the
userstable; every other table holds auser_idand joins to get the name. Change the name once, and every place that reads it is correct — that's the whole benefit, and it's why normalization is the right default for a source of truth. - Denormalization deliberately duplicates or precomputes that same fact somewhere a hot read needs it without a join — storing
author_namedirectly on eachPostrow so rendering a feed of 10,000 posts doesn't joinusers10,000 times, or storing a precomputedfollower_countonUserinstead of runningCOUNT(*)on every profile view. - The trade is real, not free: a denormalized copy can drift stale, and every write now has to update every copy (or accept that staleness for a while). Normalize first; denormalize specifically for the read you've already measured as hot.
- Normalization stores each fact exactly once. A user's display name lives only in the
- Choose the indexes. Every index speeds up reads and slows down writes. See Indexes for composite-index ordering, keyset pagination and secondary indexes on sharded data.
users (user_id PK, username UNIQUE, created_at, ...)
photos (photo_id PK, owner_id, s3_key, caption, created_at) index (owner_id, created_at DESC)
follows (follower_id, followee_id, created_at) PK (follower_id, followee_id)
+ reverse table/index keyed by followee_id
feed_items (user_id, created_at, photo_id) partition key user_id, sorted by created_at DESC
Data ownership: in a microservice design each service owns its data and others access it only through its API or events. No shared databases. That's what lets teams change schemas and scale independently. Cross-service views are built from events (CQRS / read models).
Watch for: hot keys (a celebrity's follower list), unbounded rows or partitions (one huge partition per popular entity), IDs (sortable Snowflake/ULID vs random UUIDv4, which fragments B-tree indexes), soft deletes, and multi-tenancy (a tenant ID in every key).
5. High-level architecture
Goal: a box-and-arrow diagram that serves every functional requirement end to end. Walk the write path and the read path separately.
┌──────────┐
Clients ── DNS ──▶ CDN ─┤ static │
│ │ media │◀──────────────── Object storage (S3/GCS)
│ └──────────┘ ▲
▼ │ pre-signed upload
Load balancer (L7) ──▶ API gateway (auth, rate limit, routing)
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
Photo service Feed service User/Graph service
│ │ │ │ │
│ └── publish ──▶ Message queue / log ──▶ Fan-out workers ──▶ Feed cache (Redis)
▼ ▲
Photo DB (sharded) │
Feed reads ────────────────────────┘
Cross-cutting: service discovery, config, secrets, metrics/logs/traces, CI/CD
| Component | Role | Notes |
|---|---|---|
| Clients | Web and mobile | Client-side caching, retries with backoff, offline queues |
| DNS / GeoDNS | Route to the nearest healthy region | See DNS |
| CDN | Serve static content and media from the edge, terminate TLS near users | The largest bandwidth saver. See CDN |
| Load balancer | Spread traffic, run health checks, terminate TLS | L4 vs L7, see HTTP §10 |
| API gateway | Authentication, rate limiting, request routing, aggregation | Keep business logic out of it |
| Application services | Stateless business logic | Scale horizontally with autoscaling |
| Databases | The source of truth | Choose per access pattern (component 6) |
| Caches | Hot reads | Component 7 |
| Queues / logs | Decouple, buffer, fan out | Component 9 |
| Workers | Async jobs: transcoding, fan-out, notifications, emails | Idempotent, retryable |
| Object storage | Blobs: images, video, backups, logs | Cheap, highly durable, high first-byte latency |
| External systems | Payments, email/SMS, push (APNs/FCM), third-party APIs | Wrap them with timeouts, retries, circuit breakers and webhooks |
Red flags: a single DB box serving everything at 1M QPS. Arrows with no explanation of what flows over them. Media bytes flowing through app servers. Synchronous chains of five services for one user action.
6. Database selection
Goal: pick storage by access pattern, consistency needs and scale, not by fashion. Multiple stores in one design (polyglot persistence) are normal.
| Type | Examples | Strengths | Choose when | Avoid when |
|---|---|---|---|---|
| Relational (SQL) | PostgreSQL, MySQL, Aurora; distributed: Spanner, CockroachDB, Vitess | ACID transactions, joins, constraints, flexible queries | Money, inventory, users and permissions, anything relational or transactional. The default until proven otherwise. | Massive write throughput beyond one primary, with no natural shard key |
| Key-value | DynamoDB, Redis, Riak | O(1) lookups by key, predictable latency, scales horizontally | Sessions, carts, URL mappings, feature flags, caches | Ad-hoc queries, multi-key transactions (limited support) |
| Wide-column | Cassandra, ScyllaDB, Bigtable, HBase | Huge write throughput (LSM), partition + clustering keys, multi-DC | Time-ordered data per entity: messages, feeds, IoT, activity logs | Joins, ad-hoc queries, strong multi-row transactions |
| Document | MongoDB, Couchbase, Firestore | Flexible schema, nested objects read in one go | Product catalogs, CMS, user-generated content with varied shape | Highly relational data with many cross-document joins |
| Graph | Neo4j, Neptune, JanusGraph | Multi-hop traversals | Social graphs ("friends of friends"), fraud rings, recommendations, knowledge graphs | Simple 1-hop lookups (an adjacency table in SQL/KV does those fine) |
| Time-series | Prometheus, InfluxDB, TimescaleDB, Druid | Compression, downsampling, time-window aggregations, retention | Metrics, monitoring, sensor data, financial ticks | General entity data |
| Search | Elasticsearch / OpenSearch, Solr | Inverted index, full text, fuzzy matching, facets, geo | Search boxes, log exploration, autocomplete | Being the source of truth (feed it from the primary DB via CDC) |
| Analytical (OLAP, columnar) | BigQuery, Snowflake, Redshift, ClickHouse | Scans and aggregations over billions of rows | Dashboards, BI, reporting, ML features | Low-latency transactional point reads and writes |
| Object / blob | S3, GCS, Azure Blob | Cheap, effectively unlimited, very durable | Images, video, backups, data lake files | Small, frequently updated records |
Deep dive: SQL vs NoSQL (decision framework and worked examples).
Decision questions: Do I need multi-row transactions? What are the query patterns (key lookups vs ad-hoc vs range scans vs full text vs aggregations)? Is it read- or write-heavy? How much data, and how fast does it grow? What consistency is required? Is the schema stable or evolving?
ACID vs BASE: ACID (atomic, consistent, isolated, durable) is the relational guarantee. BASE (basically available, soft state, eventually consistent) is the trade many NoSQL systems make for availability and scale. Know the isolation levels (read committed, repeatable read, serializable) and the anomalies each one prevents (dirty reads, non-repeatable reads, phantoms, write skew). Deep dive: ACID vs BASE.
Storage engines, in more depth: this is the layer below "SQL vs NoSQL" — the part of a database that actually decides how data lands on disk, and the same engine family backs very different databases.
- B-tree (PostgreSQL, MySQL/InnoDB, most traditional SQL engines) keeps data sorted on disk in a tree, so point lookups and range scans are fast — walk down the tree, or scan a sorted range. Every write has to find its place in that sorted structure and update it in place, which means random disk I/O. That's why B-tree engines are the stronger choice when reads dominate or the workload is mixed.
- LSM-tree (log-structured merge-tree — Cassandra, RocksDB, LevelDB, and underneath most "wide-column" stores) never writes in place. New writes land sequentially in an in-memory table plus a write-ahead log, then get flushed to disk as new, immutable sorted files. Sequential appends give LSM-trees very high write throughput, but a single key's current value can be scattered across several of those files, since old files are never updated — only superseded — which means a read may have to check more than one file. Bloom filters (a compact structure that can quickly say "this key is definitely not in this file") let most files be skipped, and background compaction merges old files to keep that number small, at the cost of extra disk and CPU work in the background.
Rule of thumb: B-tree when reads dominate or you need in-place consistency; LSM-tree when writes dominate — time-series, logs, feeds, metrics — and a little read-latency variance is an acceptable trade for sustained write throughput.
7. Caching strategy
Goal: serve hot reads from memory, cut latency and protect the database, without serving wrong data for too long.
Where to cache: client/browser → CDN → reverse proxy → in-process (app memory) → distributed cache (Redis/Memcached) → the database's own buffer pool.
| Pattern | Read | Write | Consistency | Risk | Use for |
|---|---|---|---|---|---|
| Cache-aside (lazy loading) | The app checks the cache. On a miss it reads the DB and populates the cache. | The app writes the DB, then deletes the cache key | Eventual. A race can leave stale data until the TTL. | The first request after a miss is slow, and there's stampede risk | The default for read-heavy workloads |
| Read-through | The cache loads from the DB itself on a miss | Same as cache-aside | Needs cache-provider support | Simplifies app code | |
| Write-through | Write to the cache and the DB synchronously | Strong for cached keys | Higher write latency, and caches data that may never be read | Data that's read right after it's written | |
| Write-behind (write-back) | Write to the cache, which flushes to the DB asynchronously | Weak | Data loss if the cache dies before flushing | Counters, likes, metrics, very write-heavy and loss-tolerant data | |
| Write-around | Write to the DB only, so the cache fills on read | Eventual | Read-after-write misses | Write-once, rarely re-read data |
Eviction: LRU (the default), LFU (stable popularity), TTL-based. Set memory limits and watch the eviction rate. Deep dive: Cache Eviction (policies, admission with W-TinyLFU, Redis maxmemory-policy).
Invalidation: "there are only two hard things in computer science..."
- Prefer delete on write over update on write. That avoids two concurrent writers leaving the older value in the cache.
- Always set a TTL as a safety net, so stale data heals on its own.
- Versioned keys (
user:42:v17) or content-hashed URLs avoid invalidation entirely. - For cross-service invalidation, consume DB change events (CDC, e.g. Debezium) to purge keys.
Failure modes to name:
| Problem | What happens | Fix |
|---|---|---|
| Stampede / thundering herd | A hot key expires and 10k requests hit the DB at once | Request coalescing / single-flight (one rebuilds while the others wait), a lock or lease, early probabilistic refresh, stale-while-revalidate |
| Penetration | Requests for keys that don't exist always miss and hit the DB (often an attack) | Cache negative results with a short TTL, or a Bloom filter of valid keys |
| Avalanche | Many keys expire together, or the cache cluster restarts | Jitter the TTLs, warm the cache, put a circuit breaker on DB fallback |
| Hot key | One key overloads one cache shard | A local in-process cache in front, replicate the key across shards (key#1..N), CDN |
| Stale read race | Reader A misses and reads the old value from the DB. Writer B updates the DB and deletes the key. A then writes the old value into the cache. | Short TTLs, leases (Facebook's memcache paper), delayed double-delete, or versioned writes |
Deep dives: Caching Strategies (read/write strategies, consistency races, stampedes, sizing) · CDN (edge caching, invalidation, origin shield). Also in this notebook: Caching Introduction, LRU, LFU.
8. Scalability
Vertical scaling (a bigger machine) is simple, has no distribution problems, and is often the right first step. But it has a ceiling, gets expensive and leaves a single point of failure. Horizontal scaling (more machines) is near-unlimited, but needs stateless services and partitioned data.
Stateless services: move session state into tokens or a shared store so any instance can serve any request. Autoscale on CPU, RPS or queue depth.
Autoscaling, in depth: it adds or removes compute capacity automatically in response to real-time demand, so you provision for the load that's actually happening instead of a fixed guess at peak. Horizontal autoscaling (add/remove instances) is the common case, since it has no single-node ceiling; vertical (resize one instance to a bigger type) is simpler but needs a restart and still has a ceiling.
- Target tracking picks one metric and a target value — "keep average CPU at 50%" — and the autoscaler continuously adjusts capacity to hold it there.
- Step scaling defines explicit thresholds ("add 2 instances past 70% CPU, add 5 past 90%") for more deliberate control over how aggressively it reacts.
- Scheduled scaling pre-scales ahead of a known spike (a sale at noon, a Tatkal booking window at 10:00 AM) because reactive autoscaling is too slow for a demand jump that happens in seconds — the metric has to cross the threshold, and only then does a new instance start booting.
- A cooldown period after each scaling action prevents "flapping": scaling up again before the last action's effect has even shown up in the metrics.
- What you scale on matters: CPU lags real user impact, request latency tracks it more directly, and for async workers the right signal is queue depth or consumer lag, not the worker's own CPU — a worker can be idle-waiting on a slow downstream call with a huge backlog still piling up behind it.
- Cold start is the real ceiling on how fast any of this can react: a freshly launched instance has to boot, deploy code, warm its caches and connections, and pass health checks before the load balancer sends it a single request. That lag — not the autoscaling policy — is why a 1,000x instantaneous spike (see Design IRCTC) needs capacity scheduled ahead of time, not autoscaled in response.
Replication: copies for availability and read scale
| Model | How it works | Pros | Cons |
|---|---|---|---|
| Single leader | All writes go to the leader, and followers replicate | Simple, no write conflicts | Write throughput limited to one node. Failover complexity. |
| Multi-leader | Several nodes (often one per region) accept writes | Local writes in every region | Conflict resolution (last-write-wins loses data. CRDTs, app-level merges.) |
| Leaderless (Dynamo-style) | Clients write to and read from N replicas with quorums | High availability, no failover | Tunable consistency, read repair, more complex semantics |
Sync vs async replication: synchronous means no data loss on failover but higher latency, and writes stop if a replica is down. Asynchronous is fast but loses the most recent writes on failover and causes replication lag. Lag breaks read-your-writes: fix it by reading from the leader for N seconds after a write, or routing a user's reads to a replica that has caught up to the user's last write (track an LSN).
Partitioning (sharding): splitting the data for write and storage scale
| Strategy | How | Pros | Cons |
|---|---|---|---|
| Range | user_id 0–1M → shard 1 | Efficient range scans | Hot spots (sequential IDs, time-based keys) |
| Hash | hash(key) mod N | Even spread | Range queries scatter. Changing N moves almost every key. |
| Consistent hashing | A hash ring with virtual nodes | Adding or removing a node moves only about 1/N of the keys | More complex. Used by Dynamo, Cassandra and CDNs. |
| Directory / lookup | A lookup service maps key → shard | Maximum flexibility, easy rebalancing | The directory itself must be highly available and fast |
| Geo / tenant | Shard by region or customer | Data residency, isolation | Uneven tenant sizes, so big tenants need splitting |
Choosing a shard key: it should have high cardinality, even access, and co-locate data that's queried together (shard by user_id so one user's posts sit on one shard). Avoid monotonically increasing keys for hash-less range sharding.
The pains of sharding (be ready to discuss them): cross-shard queries and joins (scatter-gather, or denormalize), cross-shard transactions (avoid them, or use sagas), rebalancing (move partitions, not keys, and pre-split into many logical shards), globally unique IDs (Snowflake), and hot partitions (celebrities, so salt keys like celebId#0..9 and merge on read).
Load balancing algorithms: round robin, weighted, least connections (uneven request costs), consistent hashing (cache affinity), and power of two random choices (pick 2 at random and use the less loaded one, which is great at scale).
9. Asynchronous processing
When to go async: the work is slow (video transcoding, ML), the user doesn't need to wait for it (emails, notifications, analytics), you need to smooth spikes (a queue absorbs bursts), you're fanning out one event to many consumers, or you're decoupling teams and services.
| Message queue | Log / stream | |
|---|---|---|
| Examples | SQS, RabbitMQ, ActiveMQ | Kafka, Kinesis, Pulsar, Redpanda |
| Model | A message is consumed, then deleted | An append-only log retained for days. Consumers track offsets. |
| Replay | No | Yes: reprocess from any offset, and new consumers can read history |
| Ordering | Best-effort (FIFO queues are limited) | Per partition |
| Fan-out | Per queue (use SNS + SQS for fan-out) | Many consumer groups read independently |
| Best for | Task/job queues, work distribution | Event streams, CDC, event sourcing, analytics pipelines |
Delivery semantics:
- At-most-once: may lose messages. Acceptable for metrics sampling.
- At-least-once: may duplicate. The default almost everywhere.
- Exactly-once: in practice this means at-least-once delivery + idempotent processing (dedupe on the message ID, or idempotent upserts). Kafka transactions give exactly-once within Kafka-to-Kafka pipelines only.
Patterns you should name:
- Retries with exponential backoff + jitter, with a maximum number of attempts.
- Dead-letter queue (DLQ): after N failures, park the message for inspection or replay, so one poison message can't block the queue.
- Transactional outbox: write the business row and an
outboxrow in the same DB transaction, and a relay publishes outbox rows to the broker. That solves the "DB committed but the publish failed" dual-write problem. CDC (Debezium) is a common relay. - Ordering: key messages by entity (
orderId) so all events for one entity land in one partition and stay in order. Global ordering doesn't scale. - Backpressure and consumer lag: monitor the lag. Autoscale consumers up to the partition count. Shed or degrade load when you can't keep up.
- Event-driven architecture: choreography (services react to events, loosely coupled but hard to trace) vs orchestration (a workflow engine like Temporal or Step Functions coordinates steps, which is easier to reason about and to retry).
10. Distributed systems concepts
CAP: during a network partition a system must choose between consistency (refuse or stall some requests) and availability (answer, possibly with stale data). Without a partition you can have both. PACELC adds: Else (no partition), the trade-off is latency vs consistency. That's the trade-off you make every day.
| Prioritize consistency (CP) | Prioritize availability (AP) | |
|---|---|---|
| Examples | Bank balances, inventory reservation, leader election, unique usernames | Feeds, likes counts, shopping cart (merge later), DNS |
| Systems | Spanner, etcd/ZooKeeper, HBase, single-leader SQL | Cassandra and DynamoDB (default settings), Dynamo-style stores, CDNs |
Consistency models (strongest to weakest): linearizable (it behaves like a single copy, real-time order) → sequential → causal (cause before effect, e.g. a reply after its comment) → session guarantees (read-your-writes, monotonic reads) → eventual. Pick per feature, not per system.
Quorums: with N replicas, write to W and read from R. R + W > N means every read overlaps the latest write (e.g. N=3, W=2, R=2). Lower W or R trades consistency for latency and availability.
Consensus: Raft and Paxos let a group agree on values and a single leader despite failures, as long as a majority is up (2f+1 nodes tolerate f failures). They power etcd, ZooKeeper, Consul, and the metadata layers of Kafka (KRaft) and Spanner. Use them for coordination (leader election, config, locks), not for bulk data.
Time and ordering: there's no global clock, and wall clocks drift and jump. Use Lamport clocks (a total order consistent with causality), vector clocks (detect concurrent updates), hybrid logical clocks (CockroachDB) or TrueTime (Spanner's bounded-uncertainty clocks). Never use timestamps from different machines to decide "who wrote last" without understanding the risk.
Distributed transactions:
| Approach | How | Pros | Cons |
|---|---|---|---|
| Two-phase commit (2PC) | The coordinator asks everyone to prepare, then tells them to commit | Atomic across participants | Blocking if the coordinator fails, latency, locks held across the network |
| Saga | A sequence of local transactions, with compensating actions on failure (orchestrated or choreographed) | No distributed locks, scales | Only eventual consistency. Compensations must be designed (refund instead of rollback). Intermediate states are visible. |
| Outbox + idempotent consumers | Reliable event publication between services | Simple, robust | Eventually consistent |
| Avoid it | Redraw the service boundaries so one transaction stays in one service and one DB | Simplest | Not always possible |
Other must-knows: split brain (two leaders, prevented with quorums and fencing tokens, i.e. monotonically increasing lease IDs that storage checks), distributed locks (Redis Redlock is controversial, so prefer consensus-backed locks with fencing), idempotency everywhere, and the eight fallacies of distributed computing (the network is reliable, latency is zero...).
11. Reliability and fault tolerance
Assume everything fails: disks, nodes, racks, AZs, regions, dependencies, deploys and people.
| Technique | What it does | Notes |
|---|---|---|
| Redundancy | N+1 or N+2 capacity, across multiple AZs | Remove every single point of failure (also check the DNS provider, the LB and the config store) |
| Failover | Health checks → promote a standby / re-route | Active-passive (simple, idle cost) vs active-active (more capacity, harder data consistency) |
| Timeouts | Bound how long you wait | Every network call. Shorter than the caller's timeout. Propagate deadlines. |
| Retries | Recover from transient failures | Exponential backoff + jitter, max attempts, retry budgets (for example at most 10% extra load), idempotent operations only, retry at one layer only (otherwise retries multiply: 3 layers × 3 retries = 27x) |
| Circuit breaker | Stop calling a failing dependency | Closed → open (fail fast, serve a fallback) → half-open (probe) → closed |
| Bulkheads | Isolate resources per dependency or tenant | Separate thread pools and connection pools, so one slow dependency can't drain them all |
| Rate limiting / load shedding | Protect yourself under overload | Reject early with 429/503. Prioritize critical traffic (checkout over recommendations). |
| Backpressure | Slow producers down when consumers fall behind | Bounded queues, flow control, reactive streams |
| Graceful degradation | Serve a reduced experience | Cached or stale feeds, hide non-critical widgets, read-only mode |
| Safe deploys | Most outages come from changes | Canary, blue/green, feature flags, automatic rollback on SLO burn |
| Chaos engineering | Verify resilience on purpose | Kill instances or AZs in game days (Netflix's Chaos Monkey) |
Disaster recovery: define the RPO (how much data you can lose, which sets the replication and backup frequency) and the RTO (how long you can be down, which sets the automation and standby level).
| DR strategy | RTO / RPO | Cost |
|---|---|---|
| Backup & restore | Hours / hours | $ |
| Pilot light (core data replicated, compute off) | Tens of minutes | $$ |
| Warm standby (scaled-down copy running) | Minutes | $$$ |
| Multi-site active-active | ~0 / ~0 (with sync replication) | $$$$ |
Test the restore, not just the backup. Backups need to be immutable or kept offsite to survive ransomware or a bad DELETE.
12. Security
| Area | What to cover |
|---|---|
| Authentication (who are you?) | OAuth 2.0 / OIDC for login and SSO, MFA, passkeys. Sessions (server-side store, easy to revoke) vs JWTs (stateless, fast to verify, hard to revoke, so use short expiry plus refresh tokens and a denylist). Service-to-service: mTLS or signed workload identities. See Enterprise Authentication System. |
| Authorization (what can you do?) | RBAC (roles), ABAC (attributes/policies, e.g. OPA), ReBAC (relationships, as in Google's Zanzibar: "can user X view doc Y?"). Enforce it server-side on every request. Least privilege. Check object-level access to avoid IDOR (insecure direct object reference). |
| Encryption in transit | TLS everywhere, mTLS inside, HSTS |
| Encryption at rest | Disk and DB encryption with keys in a KMS. Envelope encryption (data keys encrypted by master keys). Field-level encryption or tokenization for PII and cards. |
| Secrets | Never in code or images. Use a secrets manager (Vault, AWS Secrets Manager), rotation, short-lived credentials. |
| Rate limiting | Token bucket (allows bursts), leaky bucket (smooth output), fixed window (simple, spiky at boundaries), sliding window log or counter (accurate). Distributed counters in Redis. Limit per user, IP and API key. See Token Bucket. |
| Abuse prevention | WAF, bot detection, CAPTCHA, DDoS protection at the edge (anycast scrubbing), account-takeover detection, spam and fraud scoring pipelines |
| Input handling | Validate and sanitize. Parameterized queries (SQL injection). Output encoding (XSS). CSRF tokens or SameSite cookies. SSRF protection for URL-fetching features. |
| Data protection and compliance | PII minimization, retention and deletion (GDPR right to erasure, which must reach backups and derived data), data residency (per-region storage), audit logs |
13. Performance optimization
Method: measure → find the bottleneck (CPU, memory, disk I/O, network, lock contention, a downstream dependency) → fix it → measure again. Optimizing anything except the bottleneck is wasted effort. Use the Latency Numbers and Little's Law (concurrency = throughput × latency) to reason about it.
Latency vs throughput: latency is the time per request, throughput is requests per unit of time. Batching raises throughput but adds latency. Parallelism lowers latency until you hit a shared resource. Optimize for the NFR you actually have.
| Layer | Techniques |
|---|---|
| Database | Right indexes (composite in query order, covering indexes), read EXPLAIN plans, fix N+1 queries (batch or join), connection pooling (PgBouncer), read replicas, denormalization or materialized views for hot reads, batch writes, partitioning big tables, archiving cold data |
| Application | Async/non-blocking I/O, parallelize independent calls, avoid lock contention, object pooling, efficient serialization (Protobuf), avoid GC pressure, precompute (feeds, aggregates) |
| Network | Connection reuse (keep-alive, HTTP/2 and HTTP/3), compression (Brotli/gzip), smaller payloads (field selection, pagination), a CDN and edge compute, fewer round trips (batch APIs), co-locate chatty services in the same AZ |
| Caching | Component 7: the biggest single lever for read-heavy systems |
| Client | Lazy loading, prefetching, optimistic UI, image resizing (responsive images, WebP/AVIF) |
Tail latency: p99 is what users remember. Use hedged requests, timeouts and fewer fan-out dependencies. See Latency Numbers §6.
14. Observability
Goal: know the system is broken before users tell you, and find out why quickly.
| Pillar | What | Tools | Tips |
|---|---|---|---|
| Logs | Discrete events with context | ELK/OpenSearch, Loki, Splunk, Cloud Logging | Structured (JSON), a correlation/trace ID on every line, no PII or secrets, sampling for noisy debug logs |
| Metrics | Numeric time series | Prometheus + Grafana, Datadog, CloudWatch | Use histograms for latency (percentiles). Watch cardinality: never use user_id as a label. |
| Traces | One request's path across services (spans) | OpenTelemetry, Jaeger, Zipkin, Tempo | Propagate traceparent. Head or tail sampling. This finds which hop is slow. |
What to measure:
- Four golden signals (Google SRE): latency, traffic, errors, saturation.
- RED for services: Rate, Errors, Duration.
- USE for resources: Utilization, Saturation, Errors.
- Business metrics: sign-ups, orders/min, messages delivered. These are often the fastest outage detector.
SLI → SLO → SLA:
- SLI (indicator): what you measure, e.g. "the fraction of feed requests served successfully in under 200 ms".
- SLO (objective): the internal target, e.g. "99.9% over 28 days". The remainder (0.1%) is the error budget: while you have budget, you ship faster. When it's burnt, you freeze risky changes.
- SLA (agreement): the external contract with penalties. It's always looser than the SLO.
Alerting: alert on symptoms users feel (an SLO burn rate, for example "burning a month's budget in 1 hour"), not on every cause (CPU 80%). Every page should be actionable and link to a runbook. Use dashboards per service (golden signals) and per user journey.
15. Trade-offs and future evolution
Goal: show judgment. Every decision has a cost, so say what it is.
State decisions in this shape:
| Decision | Chosen | Alternatives | Why | What we give up |
|---|---|---|---|---|
| Feed generation | Fan-out on write, with a celebrity exception | Fan-out on read | Read-heavy (400:1), p99 under 200 ms | Write amplification, storage for the precomputed feeds |
| Primary store | Sharded Postgres | Cassandra | Transactions for follows and counts, team familiarity | Manual sharding work as we grow |
| Feed consistency | Eventual (seconds) | Strong | Availability and latency matter more | A new post may take a few seconds to appear for followers |
Review the design out loud:
- Bottlenecks: "What breaks first at 10x?" (often the primary DB's writes, a hot key, or fan-out for celebrities)
- Single points of failure: is anything left that's not redundant?
- Operational cost: can a team of 6 actually run 9 different datastores?
Cost optimization: storage tiering (hot SSD → warm → cold archive such as S3 Glacier, with lifecycle policies), CDN offload, autoscaling and spot or preemptible instances for batch work, right-sizing, compression, retention limits on logs and metrics, reserved capacity for the steady baseline, and watching egress fees (cross-region and internet egress are often the hidden big bill).
Future evolution: how the design grows. Single region → multi-region (and what that does to consistency). Monolith → services when team or scaling boundaries demand it. A new feature (search, recommendations, ML ranking) bolted on via CDC/event streams without touching the core write path. Say what you'd monitor to know when to take each step.
Reusable answer template
Copy this skeleton into every practice design:
1. Requirements FR (top 3–4, out of scope) · NFR (scale, latency p99, availability, consistency, durability)
2. Estimation DAU → QPS (avg/peak, R:W) → storage (×retention ×replication) → bandwidth → cache → servers
3. API endpoints, pagination (cursor), idempotency keys, errors, auth, async (202)
4. Data model entities, access patterns → keys, partition key, indexes, ownership
5. Architecture boxes + arrows; walk the write path and the read path
6. Databases store per access pattern + why
7. Caching what/where, pattern, TTL, invalidation, stampede/hot-key handling
8. Scale stateless tier, replication, sharding key, hot spots
9. Async queues/streams, delivery semantics, retries, DLQ, outbox
10. Distributed consistency per feature, consensus/locks, transactions (saga/2PC)
11. Reliability redundancy, failover, timeouts/retries/circuit breakers, degradation, DR (RPO/RTO)
12. Security authN/authZ, encryption, secrets, rate limiting, abuse
13. Performance bottleneck, latency budget, DB and network optimizations, tail latency
14. Observability golden signals, logs/metrics/traces, SLOs, alerting
15. Trade-offs decision table, what breaks at 10x, cost, evolution
Worked designs on this site
Each one applies this blueprint to a different kind of problem:
| Design | Main deep-dive components |
|---|---|
| Design Instagram | 5, 7, 8 (feed fan-out, media storage, CDN) |
| Design URL Shortener | 2, 4, 7 (ID generation, caching, redirects) |
| Design AWS EC2 | 5, 10, 11 (control plane vs data plane, placement, isolation) |
| Design Uber | 4, 8, 9 (geo-indexing, real-time location, matching) |
| Design Email Campaign System | 9, 11, 12 (bulk async sending, retries, abuse) |
| Design Invoice Follow-up System | 9, 10 (scheduling, workflows, idempotency) |
| Design Enterprise Authentication System | 3, 12 (SSO, tokens, multi-tenancy) |
Final red flags checklist
- No numbers on the NFRs, and no estimation-driven decisions
- Jumping to microservices, Kafka and Kubernetes for 40 QPS
- A single database for everything at massive scale, with no sharding story
- No idempotency on retried writes (payments, orders, messages)
- Caching with no invalidation or TTL plan
- Ignoring failure: no timeouts, retries, redundancy or degradation
- "Strong consistency everywhere" or "eventual consistency everywhere" without tying it to features
- Never stating a trade-off